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ABSTRACT 

An observational monitoring system is described for 
evaluating preschool programs and providing feedback to relatively 
untrained caregivers in order to encourage program improvement in the 
social and cognitive realms. The system utilizes a simple one page 
observation form for teachers and children detailing interaction and 
type of activity. Data are collected through accumulation of 
instantaneous observations with high interobserver reliability. 
Results show strong differences among three distinctive urban 
preschool settings, especially in teachers' behavior and children's 
sociometric patterns. The system's stress on simplicity and feedback 
should serve as a widely applicable formative evaluation tool for day 
care and nursery school settings. (Author) 
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AbSTIlACT 



An Obscrvntional System Cor tlic Hvaluation of l^reschool Programs: 

l^csearcli nnd Dovclopmenl 

Shirley S. Angrist, Carncgic-Mcllon University 

llelcne liorko, Hay Associates 

This paper deacribcii the pilot study of an ongolnc effort to develop an 
objjervational monitoring system which can tap proj^ram quality in preschool 
settings, including nursery lichools and day care centers. The research ob- 
Jcctivo is to devise simple techniques usable by t«'achers or carefjivers for 
ar,se93inp, tlie level of social and cognitive cunl hul and for providing feed- 
back to teaciiors so that they can alter and improve pro.^rams. The monitoring 
sy:jtcm consiiits of one pn^e forms Cor observin;; children ami teach<u-s in tems 
oi." Intt^rac t ion, intent of the interaction, and the extent oT cognitive, lanC'.uage, 
creative or motor activity. In addition, the system Includes a manual for 
traininji observers and a resource book of pro:vaui materials for teachers. The 
observ.i t ions were collected over six weeks throu'.h an accumulation of instantaneous 
photogcapliic-like pictviros for eacli child and tuachcr. This method yields 
h^{]h V'^Ti ability v;j.th lul er-obacrvr r a!;reeiiient i.iLeb of over 90 percent. 
The data based on 20 obsc^rvat Lons per ciiild and 30 per teacher were analyzed 
using analysis of variance and canonical analysis to ascertain school differ- 
ences. Tlie observational system was tested in 1 urban preschool settings. 
The results show that the three scliools, while s.UTi.llar in fosteiin^^ construc- 
tive activ^ity, differ both in teacMer and child l)eh.wiors. Significant difier- 
ences occur In th2 extent of teaclu^rs* IntLTaction v/lth children, prosocial 
beliavior and creativity, as well as in the amount oC children's involvement, 
interaction with other children, cognitive and Line motor activity. This 
monitoring system, while still under development, is important in answering 
questions about the situational and or|;anlzed contexts which foster social 
and intellectual competence in children. The implications of this research 
are twofold: (1) providing effective methods Cor monitoring the environmental 
dimensions of early childhood educational programs, so that caregivers and 
parents can understand the process and implement desired changes in the on- 
going system; and (2) training caregivers to create and sustain intellectually 
stimulating and socially constructive child care environments. 



The growing Interest In early childhood education outside the family 
reflects the simultaneous recognition that lamlly functions arc changing 
In the U.S. and that developmental environments are necessary for young 
children. In addition to programs which emphasize play and social develop-* 
ment there are a number of programs specifically designed for educating 
preschool children. These Include the Montessorl method, Piagetlati-baaed 
learning, the open classroom, and Berelter and Engleman's skills 
development approach. One of the major questions sLlIl to be answered is 
what effect these various programs have on the children attending them. 

How can we measure what happens in preschool settings? Can we gauge 
the educational quality of the environment and discriminate good programs 
from bad ones? The purpose of the rosearch reported in this paper is to 
develop an observational monitoring system which can tap program quality 
(regardless of orientation) in preschool nettings. Our aim is to devise 
simple techniques usable by outside observers, teachers or caregivers, for 
assessing the level of social and cof;nitive functioning and for providing 
feedback to teachers so that they can altrr and improve programs. 

One of the main strategies emphasi^ccl by educational researchers has 
been to assr!ss knowledge of subject matter, acquired as a result of a par- 
ticular educational intervention using measures such as reading readiness or 
school achitivement . Such approaches have benn critized recently because they 
use sunmiative measures rather than formative ones (l>loom, 1971) and because 
they frequently focus on inappropriate (limrnsions of program quality (Shapiro 
1973). Mearuires which dwell on school achievement ignore social and inter- 
personal competencies, as well as creative and problem-solving skills. In 
addition, achievemeni: scores available **af i:cr-the-f act" , are unasuable for 
changing an ongoing program. Bloom (1971) advocates a formative approach 
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using a voriety of measures to evaluate actual classroom activities. Especially 
in preschool settings, formative evaluation may be the mechod of choice in 
order to provide feedback to teachers (Kamii, 1971; Shapiro, 1973). 

The concern with evaluation in early childhood education is reflected 
in the proliferation of observation systems for the preschool and primary 
elementary school levels. The Analysis of Communication in Education 
(Bovnnan, 1972), The Differentiated Child Behavior Observation Classroom Scan 
(Ross and Zimiles, 1973), as well as the set of observational instruments 
developed at the Stanford Research Institute (Stallings et. al., 1973) reflect 
efforts to assess Follow Through programs, the sequel to Head Start. Other 
observational systciTis, like PROSE (Medley ct. al., 1971) grew out of the 
concern to syst cmiit i.cally describe the cla.ssroom context. 

The varie«l observational instruments utilize a mixture o* methodologies: 
some concentrate on classroom climate as veflccted in teachers* behavior 
(Withall, 1949), others focus primarily on teachers in relation to children 
(Medley et. al., 1971 and Firandt, 1972), .still others try to picture the whole 
room or situation (Koss and ZimLles, 1973). Observational techniques include 
instantaneous nnapshot-likc metliods and longer time scans. 

Our system builds* on existing instrumonLs but with the aim of creating 
a flexible system v/liich is easy to use, yet one which highlights the key 
elements of *'adequnce" preschool programs. We began with Kamii's (1971) 
assumptions that a f,ood program should coniain social, emotional, cognitive, 
perceptual-motor and creative componcni s . Ilrnce our instruments asness social 
behavior in interactions, participation pai tcrUvS, the use of language > 
mathematical and general knowledge, fine and gross motor coordination, and 
rreative activities. 

ERLC 
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In addition wr tiought a system (1) flexible enough to assess situational 
or group context, on the one hand, and individual children or teachers' 
behavior, on the other hand; (2) capable of focusing predominantly on children 
o£ on teachers through parallel but separate forms; (3) yielding high reliabil- 
ity; (4) producing easily computerizable data; (5) and simple enough to be 
used by teachers for their own feedback. 

The Setting 

The observational system was tested in three urban preschool settings. 
The three schools represent different sponsorship, socioeconomic levels and 
program philosoj Uy. 

School T If n nelf ouppoitiiig uiiivertiicy based day care center. Fees 
are charged on r sliding scale. The program is ociented around play and 
social developrrc it The 31 youngsters attending the school are predominantly 
children of univ^rsLty students bvit also ol' staff, faculty and community 
people. The grc ip is mainly white with 25/« from aonwhite backgrounds. The 
children a\e divided into 2 groups according to af^,e; each group has about 
2/3 boys and 1/3 girls. The ratio of children to teachers, is 15 to one. 
If aides are included the child-adult ratid is 8 to one. 

School II i.i an OEO subsidized day care center. The program combines 
free play and. thf use ol: structured kits to teach cognitive skills. Only 
children of poverty families are eligible to attend under Title IV A. Tlie 
30 children are separated into two classrooms with 3 and 4 year olds in each 
room. All the youngsters are black and 607, are boys. As in School I, the 
child-teacher ratio is 15 to one. If aides are included the child to adult 
ratio is 8 to ono. 
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School 3 i» a university sponsored preschool and kindergarten. Fees 
are comparable to those of other private nursery schools. The program 
emphasizes an open classroom individuali^.cd approach and encourages cognitive 
growth. One fourth of the children are youngsters of faculty membera vith 
the remainder coming from the general coimnunity . The 3, U and 5 year olds 
are all combined into one large group of approximately 30 youngsters i Half 
the children are boys » half girls and .157. are from non-white backgrounds* 
The child-teacher ratio is 10 to 1 but drops to about 5 to 1 when students 
function as teaching aides « 

Procedures 

Tlia monitoring system involves four omponents; observations forms 
lor teachers, fcrms for children, a manual, for instructing observers, and 
resource materials for use by teachers. hose components may be used 
separately or in combination. Copies of lie observation forms and the manual 
are available from the authors. 

Each form provides data on internet! wi, including who interacts with 
whom, the intenl. of the interaction, and type of activity in terms of language 
or other cognitive, creative or motor conlcnt. To collect the data, the 
observer first [;et3 acclimated. Then acting like a photographer at the 
sound oC a signal from the tape recor<ler, thn observer notes v;hat the observed 
child or teacher is 6oinf\ at that instant an«t th:in coiiiplctes the form. Observers 
randomly select which child or teacher to obscrvo until all in the school have 
been observed at least once before maklnr, x secoad observation. Observations 
were made over two months in tliroe scliooLs until we had 30 observations per 
teacher and 20 per child, All obr>orvati (^n;; we^'o made during the morning 

Two trained obs'-irvers observing about 3 hours a day can complete about 2 
observations foe each of 30 chiltiren and ibi^ut ^ observations on each of 3 
teachers. We found tliat with iibscnccs, class trips and other special 

Q activities, the accumulation of COO observaiions for children and 100 for 

JC teachers required at least 3 to 6 weeks in each school* 
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since In most programtt this la the key Indoor program time. Morning and Indoor 
activity seemed to reveal best the programmatic emphasis of the school: 
activities were usually otganited by the teachers, or free play or transitional. 
At other times of the day more time was spent in meals, naps and outdoor 
play. The data reported in this paper stem from observations of individual 
children and teachers. 

Reliability was assessed in two ways: interobserver agreement and 
stability over time. For both children and teachers, observers worked in 
pairs. A realiabllity check was made on every tenth obs^ervation; observers 
synchronized these observations to respond to the same sound signal and the 
same subject in the same context. The interobserver agreement on observational 
categories for teachers' behavior was 97 percent, for children, 95 percent* 

Stability of behavior over time was assessed throu[;h comparing tv/o sets 
or teacher observations: 30 observations taken over a 4-6 week period were 
compared wLth 20 additional observations per teacher taken in one day. Ten 
out of 12 chi square tests of the distributions of teacher behaviors under 
each condition were not statistically significant. Hence we conclude that 
the teacher form yields rather stable patterns over time. 

Mc reviewed the validity of the behavior categories in several way9« 
First, we modified or eliminated categories which, even after careful 
training, observers interpreted differently. Second, wo tested to see 
whether other variables such as sex or age might strongly affect the 
bcliavioral categonics. There were no differences at all between boys and 
girls in any of the schools on any of the categories. In addition, we 
checked for the possibility that School III children might score higher because 
some of them are five years old, while the other two schools have only three and 
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four year olds. Comparison of the five year olds with the three's and 
four's in School III yielded no significance differences in chi square 
tests of the in»:idence of each behavior category. Finally, we presented 
teachers in two schools with observational results based on earlier versions 
of the instrutncuts and modified or eliminated behavioral categories according 
to their reactions about confusing, unclear or invalid data. 

Results 

Our hypoth'sis that the monitoring instruments should detect differences 
in progranmiatic orientation amonr; the three prcschools was generally borne 
out, but with s >me qualif icalion.o. 

First, we iiscovered some surprising t;imilarities among the schools. 
Children were c nstructively involved in activities 90 percent of the time 
Uhey were obser ed. Rarely (about 10 percent of the time) were children 
in any of the sciooJs found to be bored, sad, angry or crying. Overall 
interaction rate 5 among r.dulLs and children were fairly similar children 
were observed ir interaci ions of some kind about half the t5.me. Perhaps 
because the datr were co lectcd indoors, gross motor activity represented 
only about 15 percent o the morning's activities in each school. 

Even for toachnrs, (here were some school similarities: disciplining 
oi: children by ostracism, shoutinr; or physical punishment was >-are. But 
also rare were efforts to clarify social inueraction or to express affection. 

In order to test the hypotlic'us that the three schools should differ 
in teacher and child beht vior, wt? used onalysif\ of variance and canonical 
correlation tecimiques. For Che analyses of variance we treated each 
observation as a discrct( unit of data and converted each observation to 



7 



a score from 0-2, with 0, if the event did not occur, 1, if the event 

occurred once, and 2, if the event occurred twice per observation. 

Table 1 presents the sumnary of the analysis of variance results for 

it 

the three teachers in each school (with 30 observations per teacher) . 
On all tho teacher variables, School III scores higher than School II, and 
School II scores higher than School I. School III teachers Interact with 
children, express prosoclal intent in their interactions with children and 
are involved in creative activity with children significantly more than the 
teachers in Schools I and II, School Til teachers are also highest on cogni- 
tive and gross motor activity, but the.'ie di Tfereaces do not nahicve statis- 
tical significance. 

VJhcn we turn to the child behavior variables presented in Table 2, it 
is evident that School 1 children tend to score higher than School II children 
in all catej^orios except tor Clue motor activity. This confirms what our 
observers noted that School II was well supplied with puzzles compared with 
School I which had few toys rc'tuiring fine motor skills. In contrast with 
the other tv;o schools, School i.II children score highest on iiil behaviors. 
They are significantly more in^rolved overall, they interact vith other 
children more, ihey exhibit more prosocial behavior, and they spend more time 
in cognitive activity. 

To determine the extent to which the schools differ in overall programrning, 
VG US :d canonical analysis with the computer statistical package developed 

if 

For all schools we omitted observations on aides from these analyses 
because their roles and numbers vary {;rcatly, and because we assumed 
that teachers are the main determiners of the school program. 



8 



by GlMton (1973). Only the behavioral categories which reached statistical 
significance in the analysed of variance were included • Observations on each 
behavior category were sunned separately for teachers and children for each 
morning of the data collection period. The bulk of the data was collected 
in IS days at each school. It is these 15 sets of 'Wrning observation scores'* 
which served as the input for the canonical analysis. 

The canonical analysis yields two variates. Table 3 presents the struc- 
ture matrix for each variate. These ore the correlations of each behavioral 
and school variable, to each variate and are interpretable as factor loadings 
in factor analysis (Stewart and Love, 1968). The first variate is dramatically 
most characteristic of School III and least characteristic of School t. School 
III is high on children's participation in cognitive activity. It is also 
high on the amount of teacher interaction with children, on teachers* ex- 
pression of prosocJal intent and on teachers' encouragement of creativity. 
Variate B best desr ribcs School II as very hi;;h and School I as extremely 
low in children's fine motor activity. With 50 percent of the variance in 
schools explained hy the behavioral categories, the results clearly suggest 
important school differences in program content. It is interesting to note 
that in School Til both children's and teacher's behaviors serve to highlight 
program dif f erencer, . 

Figure 1 presents the canonicil analysis results graphically, showing 
the extent to whicli the behavioral variables occupy the position in space 
covered by the Jjchnols. With Variaie A on the horizontal axis and Variate 
B on the vertical axis, we can sec that Sciiool III dominates the graph 
surrounded by the cluster of both child and teacher variables. Seven of the 
sight variables clearly characterize School III and tend to be diametrically 



opposite descriptors of Schools I nnd II. The teacher variables are closest 
to School III; the first three child variables arc also nearest to School III 
but they are somewhat closer than any other variables to School I. School II 
is in reasonable proximity to variable 3» children's fine motor activity. 
Schools I and II appear rather isolated spatially from School III. We can 
infer that in School III, the teachers* cmphasin on Interaction with children 
and their manifestation of prosocial behavior encourages the children to 
interact and to express prosocial intent. Fisure 1 confirms the greater 
developmental quality of School Ill's program. 

Discussion 

The resu'ta of the observational data presented so far indicate that 
i,\Q observation system does tap empirical differences in schools which are 
c larac terized an having; different programs. While this fulfills our primary 
;.>al to devel op simple observational instruments for assessing school quality, 
tiicre arc oth »r important consideration:; which justify these instruments as 
part of o mon torinfi system. 

Ideally^ vAwn it is fully opcrativ^^, the system makes possible the following: 

(1) CeedUack to teachers about their own behavior as individuals or 
as a };;roup. 

(2) feedback to teachers about children either as individuals or as 

a grdup. 

(J) a pirturci of chanRes in tcnchcrr;' or children's behavior over 
t imo » or due to pto^vam altcrai ions. 

Once the obs* rvers are trained and Ihr observational data arc collected, 
key punched and stored on a computer, tl)c ovaluators can feed back information 
about each behavior cni.egory using data derived from simple frequency distribu- 
tions . 
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Since this research bcQan^ we have had the opportunity to deter- 
mine the instruments' ability to detect program changes within a school. 
By chance > two of the three schools underwent significant alterations in 
staffing! Blze and equipment. Our preliminary results indicate that the 
instrunients do reveal changes in the behavior categories consequent to 
these program alterations. 

In a preliminary trial of the system, the Investigators stored the data 
in an SPSS fileCNie, Bent and Hull, 1970) generated marginal distributions 
for each behavior category and then presented the results for individual 
children and for the whole school to teachers. Figure 2 has examples of the 
visually attractive and quickly interprctablc results on two behavior cate- 
gories as they were presented to teachers. Each circle represents 100 
percent or all observations in that brhavior caiegcry. The darkened area 
indicates the percent of occurrence of t\u\ ;;ivcn behavior. The teachers 
reacted by coimicnting about tUt! individual ciiildren and by reassessing some 
children's behavior. We strongly recoiwncnd /iO observations per subject when 
providing teachers with feedback on an individual. 

In order to reduce the time needed to collf?ct observations in a school 

setting, wo arc extending the monitoring syr>tom Ny developing {;roup or situationally 
oriented observation fonns. With the situation observation forms, observers 
count the numb-r of children in ench activity c.itegory through a series of 
successive instantaneous observations, Ohri ^rvatlons in one classroom can 
ho ancunulatcd rapidly and reduced by coinpui or to provide fastnr feedback to 
teachers. Since the situation forma, as well as the individual child and 
t:cft i r forms center on the same acti^dty dimensions, the system can serve 
in formative evaluation in two key ways: 



u 

(1) for contextual aasesBment of school program In terma of predominant 
co(^nltlve and aoclal actlvltlca Involving children and teachers 
and 

(Z) for behavioral aaoeosi^vent of individual teachero or childten in 
order to train teachers in dealing with specific children or 
for altering the teacher's own behavior. 

Conclusion 

Although our efforts to develop and refine the monitoring system are 
far from complete, wo arc encouraged by the results reported in this paper. 
The system's asf^cts lie in its ability to detect the crucial cognitive and 
prosocial components of p;Gnerally good but rather diverse preschool programs, 
its applicability for reflecting individual as well as overall school or 
classroom bt^havior, and its adaptability for feedback to teachers and pro{;ram 
planners. The system's primary limitation lies in the time delay between the 
o>)servational data collection to the feedback stage. A computerized informa- 
tion system could easily be developed to take the precoded observation sh(!et8» 
score them by optical scanner and print out summary results. 
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TABLE 1* Staniatry of Analysea of Variance for Observaclons on Teachtr Bahaviora 



School Mean Score 



Variable 



I 

(N-90) 



II 
(N-90) 



III 
(N-90) 



F-Ratlo 



Interacts with children 


.54 


.61 


.81 


8.05** 


Shows prosocial intent 


.14 


.21 


.37 


6.65** 


In cognitive activity 


.16 


.17 


.29 


2.81 


In creative activity 


.02 


.08 


.14 


4.61* 


In fino motor activity 


.07 


.10 


.14 


1.47 



Differences /imong schools are statistically significant at p > .05 
Differences among schools are significant at p > .01 
For all analyses, df « 2/267. 
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TABLE 2, Simnary bf AMlyses of Variance for Observations on Child Behaviors 



School Mean Score 



Variable 


I 

(N»545) 


II 
(N=600) 


III 
(N-555) 


F-ratio 


Child is involved 


.92 


.91 


i95 


3.25'V 


Interacts with children 


.29 


.21 


.29 


5.92** 


Interacts with adult 


.23 


.20 


.25 


1.97 


Shows prosocial intent 


.33 


.27 


.38 




In cognitive activity 


.19 


.13 


.28 


.^.Ub-'-* 


In creative activity 


.18 


.16 


.18 


.92 


In fine motor activity 


.14 


.28 


.23 


16.43** 



Differences among schools are statistically significant at p > «05 
Differences among schools are significant at p > .01 
For all analyses, df * 2/1697. 
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TABLE 3* Results of Canonical Analysis 



Structure Matrix of Canonical Co«fficisnta 
Variate A Varlate B 



For Behavioral Categories i 



!• Child is involved 0.25 -0.19 

2. Child interacts with children 0.10 -0.19 

3. Child shows prosocial intent 0.27 -0.27 

4. Child in cognitive activity 0.54 -0.12 

5. Child in fine motor activity -0.09 0.79 

6. Teacher interacts with children 0.73 -0.01 

7. Teacher shows prosocial intent 0.69 0.35 

8. Teacher encourages creativity 0.46 0.34 



For Schools: 

School I -0.26 -0.97 

School II -0.71 0.71 

School III 0.96 0.26 



R , = 0.81 

cl 



c2 
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FIGURE 1. Program Characteristics of Three Schools: 

Structure Matrix for TWo Canonical Varlates 



+ 1 

VARIATE B 



School JI 



s. 



1 



School I 



Variables : 



8« 7. 

School HI' 



^1 



2. 1 



1 = child Is Involved 

2 « child interacts with children 

3 " child shows prosoclal intent 

4 ■ child in cognitive activity 



5 = child in fine motor activity 

6 = teacher interacts wl»-.h children 

7 = teacher shows proaoclal Intent 

8 = teacher encourages creativity 
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FIGURE 2. Graphic Presentation of Observations in Selected Behavior 
Categories for Individual Children and for Schools. 
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